Project Overview

  • Goal: Construct a model that accurately predicts whether an individual makes more than 50k/yr
  • Motivation: Help a non-profit identify donor candidates and understand how large of a donation to request
  • Data Source: 1994 US Census Data UCI Machine Learning Repository*

Note: Datset donated by Ron Kohavi and Barry Becker, from the article "Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid". Small changes to the dataset have been made, such as removing the 'fnlwgt' feature and records with missing or ill-formatted entries.

Table of Contents:

  1. Exploratory Data Analysis
  2. Data Preprocessing

Exploratory Data Analysis (EDA)

Featureset Exploration

  • age: continuous.
  • workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
  • education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
  • education-num: continuous.
  • marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.
  • occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
  • relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
  • race: Black, White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other.
  • sex: Female, Male.
  • capital-gain: continuous.
  • capital-loss: continuous.
  • hours-per-week: continuous.
  • native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 45222 entries, 0 to 45221
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   age              45222 non-null  int64  
 1   workclass        45222 non-null  object 
 2   education_level  45222 non-null  object 
 3   education-num    45222 non-null  float64
 4   marital-status   45222 non-null  object 
 5   occupation       45222 non-null  object 
 6   relationship     45222 non-null  object 
 7   race             45222 non-null  object 
 8   sex              45222 non-null  object 
 9   capital-gain     45222 non-null  float64
 10  capital-loss     45222 non-null  float64
 11  hours-per-week   45222 non-null  float64
 12  native-country   45222 non-null  object 
 13  income           45222 non-null  object 
dtypes: float64(4), int64(1), object(9)
memory usage: 4.8+ MB
Out[4]:
age workclass education_level education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country income
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Out[5]:
count unique top freq mean std min 25% 50% 75% max
age 45222 NaN NaN NaN 38.5479 13.2179 17 28 37 47 90
workclass 45222 7 Private 33307 NaN NaN NaN NaN NaN NaN NaN
education_level 45222 16 HS-grad 14783 NaN NaN NaN NaN NaN NaN NaN
education-num 45222 NaN NaN NaN 10.1185 2.55288 1 9 10 13 16
marital-status 45222 7 Married-civ-spouse 21055 NaN NaN NaN NaN NaN NaN NaN
occupation 45222 14 Craft-repair 6020 NaN NaN NaN NaN NaN NaN NaN
relationship 45222 6 Husband 18666 NaN NaN NaN NaN NaN NaN NaN
race 45222 5 White 38903 NaN NaN NaN NaN NaN NaN NaN
sex 45222 2 Male 30527 NaN NaN NaN NaN NaN NaN NaN
capital-gain 45222 NaN NaN NaN 1101.43 7506.43 0 0 0 0 99999
capital-loss 45222 NaN NaN NaN 88.5954 404.956 0 0 0 0 4356
hours-per-week 45222 NaN NaN NaN 40.938 12.0075 1 40 40 45 99
native-country 45222 41 United-States 41292 NaN NaN NaN NaN NaN NaN NaN
income 45222 2 <=50K 34014 NaN NaN NaN NaN NaN NaN NaN 
Number of observations: 45222
Number of people with income > 50k: 11208
Number of people with income <= 50k: 34014
Percent of people with income > 50k: 24.78

Data Preprocessing

Before this data can be used for modeling and application to machine learning algorithms, it must be cleaned, formatted, and structured.

Split the data into features and labels

Skew

The features capital-gain and capital-loss are positively skewed (i.e. have a long tail in the positive direction).

To reduce this skew, a logarithmic transformation, $\tilde x = \ln\left(x\right)$, can be applied. This transformation will reduce the amount of variance and pull the mean closer to the center of the distribution.

Why does this matter: The sample being observed may be a poor aproximation of the population.

Asymptotic normality indicates that a sample will approach a normal distribution even if the population is skewed.

  • As per the central limit theorem, as the sample size (the number of means, i.e. the number of samples) increases, $n\rightarrow \infty$, the sampling distribution of the means will become more normally distributed even though the population distribution is skewed.
  • When $n$ is large, asymptotic theory provides us with a more complete picture of the “accuracy” of $\lambda$: By the Law of Large Numbers, $\bar{X}$ converges to $\lambda$ in probability as $n \rightarrow \infty$. Furthermore, by the Central Limit Theorem, $$n(\bar{X} − \lambda) \rightarrow N (0, Var[X_{i}]) = N (0, \lambda)$$ in distribution as $n \rightarrow \infty$. So for large $n$, we expect $\hat{\lambda}$ to be close to $\lambda$, and the sampling distribution of $\hat{\lambda}$ is approximately $N\left(\lambda, \frac{\lambda}{n}\right)$

Source:Stanford: Introduction to Statistical Inference

Feature Skewness Mean Variance
0 Capital Loss 4.516154 88.595418 1.639858e+05
1 Capital Gain 11.788611 1101.430344 5.634525e+07

Apply the logarithmic transformation:

Feature Skewness Mean Variance
0 Capital Loss 4.516154 88.595418 163985.81018
1 Capital Gain 11.788611 1101.430344 56345246.60482
2 Log Capital Loss 4.271053 0.355489 2.54688
3 Log Capital Gain 3.082284 0.740759 6.08362

The logarithmic transformation reduced the skew and the variance of each factor.

Feature Skewness Mean Variance
Capital Loss 4.516154 88.595418 163985.81018
Capital Gain 11.788611 1101.430344 56345246.60482
Log Capital Loss 4.271053 0.355489 2.54688
Log Capital Gain 3.082284 0.740759 6.08362 
[NbConvertApp] Converting notebook Polishing_Donor_Classification.ipynb to html
[NbConvertApp] Writing 4675068 bytes to WIP_Class_Code.html
[NbConvertApp] Converting notebook Polishing_Donor_Classification.ipynb to html
[NbConvertApp] Writing 4642221 bytes to WIP_Class_No_Code.html
[NbConvertApp] Converting notebook Polishing_Donor_Classification.ipynb to slides
[NbConvertApp] Writing 4646670 bytes to WIP_Class_No_Code_Slides.slides.html